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ABSTRACT 

California's College of the Canyons has used the 
College Board Assessment and Placement Services (APS) test to assess 
students' abilities in basic and college English since spring 1993. 
These two reports summarize data from a May 1994 study of the 
predictive validity of the APS writing and reading tests and a June 
1994 effort to validate the cut scores for the writing test. The 
first report indicates that, based on the observed relationships 
between test scores and subsequent academic performance for 
first-time students in spring and fall 1993, the median predictive 
validity value of the writing test across all levels of English 
courses wa. f .34, a moderately good relationship and enough to justify 
its use for placement. This report also indicates that for the 
reading test, the median predictive validity value across all levels 
was .14, a low linear relationship which does not warrant the test's 
sole use for placement. The second report describes the method used 
for establishing placement cut scores; i.e., determining the 
proportion of students with given scores who received passing and 
failing grades scores to determine the scores' ability to predict 
success. The report then describes an analysis of course grades for 
340 students in spring and fall 1993 placed according to their 
writing test scores, indicating that more than one set of possible 
cut scores resulted* Finally, the report indicates that recommended 
cut scores were set in the lower part of the optimum range and were 
found to be valid predictors. (KP) 



it it it it it it it it it it it it it it it it it it it it it it * it it it it it it it it it it * it it it it it it it it it it it ft it it it it it it it it it it it it it it it it it it it it it it it 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 



y— 

S 

O 

Q 
LU 



Predictive Validity Study of the APS Writing and Reading Tests [and] 
Validating Placement Rules for the APS Writing Tests 



■ PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

N. Mattice 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 



US DEPARTMENT OF EDUCATION 

OH»ce of Educational Researcn and improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 
T his document has Deen reproduced as 
'ece'ved 'rom the person c organization 
originating it 

Minor changes have been made to improve 
'^production quality 



• Points Of view or opinions slated m Ihis dOCu 
ment do not necessarily represent official 
OEHI position or policy 



Office of Institutional Development 
College of the Canyons, Valencia, CA 



0 



cr 




BEST COPY AVAILABLE 



CoMege of the Canyons 
Santa Clarita Community College District 



PREDICTIVE VALIDITY STUDY 
OF THE 

APS WRITING AND READING TESTS 



May 1994 



Office of Institutional Development 



MATRICULATION 



PREDICTIVE VALIDITY STUDY OF COLLEGE BOARD ASSESSMENT AND 
PLACEMENT SERVICES FOR COMMUNITY COLLEGES 
WRITING AND READING TESTS 

MAY 1994 

This report summarizes the predictive validity studies 
carried out in May 1994. The College Board APS Writing 
Test, a 40-question objective format test, was examined to 
determine its ability to predict student success in both 
basic skills and college level English courses offered at 
College of the Canyons. Similar examinations were carried 
out for the College Board APS Reading Test, a 35-item 
objective format test. 

The college began using the College Board APS tests in 
Spring 1993 and the data for these computations represent 
information from first-time students for two semesters 
(Spring 1993 and Fall 1993). 

Matriculation regulations require that community 
colleges which adopt a standardized test for making 
placement recommendations must qualify its use locally. The 
test must be a valid predictor of future course success 
given the unique features of the college curriculum, its 
instructors, and student population. Establishing a clear 
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relationship between the test score and subsequent student 
performance in a particular course for which placement 
recommendations were made is predictive validity. A 
correlation of no less than +.35 is needed to conclude that 
the test instrument has sufficient predictive validity for 
advisement purposes. 

Descriptive Statistics . The data collected for this study 
are summarized in Table 1. The mean score and the 
coefficient of skewness provide a clue about the relative 
degree of difficulty of the tests. Positively skewed score 
distributions indicate that sampled students tend to find 
the test difficult. Thus, the APS Writing Test was clearly 
more difficult for students in English 035 than was the test 
for students in English 101. In a similar fashion, the APS 
Reading Test was more difficult for students in English 034 
than it was for students in English 101. 
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Table 1 . 

Descriptive Statistics for APS Reading and Writing Tests , 

By Course Level 



Test 


Course 


N 


Mean 


Median 


Standard 
Deviation 


Skewness 


Writing 


Engl 


Oil 


6 


12.33 


10. 00 


5.32 


0.46 




Engl 


035 


4 ^ 


1 5 4 2 


1 4 OO 


4 . 04 


1 69 




Engl 


090 


250 


22.70 


23.00 


3.54 


-0.30 




Engl 


101 


39 


29.49 


30. 00 


4 .83 


-0.89 


Reading 


Engl 


010 


10 


8.00 


8.50 


2.79 


-0.83 




Engl 


034 


77 


13.94 


14.00 


2.93 


1 .35 




Engl 


080 


67 


18.97 


19.00 


3.41 


-0.24 




Engl 


101 


39 


26.18 


27.00 


3.63 


-1 .00 
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Predictive Validity. The primary index of the strength of 
the relationship between test scores (predictor) and course 
grades (criterion) is the predictive validity coefficient, 
or the correlation between predictor and criterion. 

Two correlation coefficients indicating predictive 
validity were computed for each course — one between the 
test scores and course grades with all withdrawals (Ws) 
converted to Fail (F), and the second between the test 
scores and course grades with withdrawals deleted from the 
analysis. The resulting correlations are shown in Table 2 
on the following page. Since the distribution of scores was 
restricted because the placement test scores were ir use, 
the correlation values were adjusted by using a correction 
for restriction or range. The correction yields a better 
indication of the relationship between scores and grades by 
adjusting the value of the correlation coefficient to what 
it would be if the distribution of the predictor (APS test 
score) was unrestricted. 
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Table 2. 

Relationship Between APS Reading and Writing Test Scores 







and 


Grades , By 


Course 




Test 


Course 


r-Score 
W=Failure 


to Grade 

Ws Excluded 


Writing 


Engl 


Oil 


.56 


( 6) 


.38 


( 5) 




Engl 


035 


.37 


( 45) 


.44 


( 36) 




Engl 


090 


.30 


(244) 


.37 


(212) 




Engl 


101 


.26 


( 33) 


.16 


( 30) 


Reading 


■ Engl 


010 


. 86 


(10) 


.96 


( 7) 




Engl 


034 


.44 


(77) 


.41 


(62) 




Engl 


080 


.11 


(67) 


.17 


(59) 




Engl 


101 


-.07 


(38) 


-.61 


(30) 
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RESULTS AND DISCUSSION 
APS Writing Test , With the exception of English Oil - 
Grammar and Writing , that had only a maximum of six students 
in the correlations, the median predictive validity value 
across all levels is .34, ranging from a low of .16 to a 
high of .44. The correlation indicates a moderate linear 
relationship between the placement test scores and course 
grades . 

The predictive validity coefficient value of .34 is 
only slightly below the .35 level recommended and is within 
the range reasonably expected for placement tests. There is 
a broad range of factors that are not, and can not, be 
tapped by the test, but which are related to performance in 
class. Among these are an assortment of individual 
difference variables, motivation level, and differences in 
background and experiences. Because there are such a 
complex array of factors that interact to determine 
performance in classes, many of which are non-academic, it 
is not reasonable to expect placement tests to be very 
strongly related to course grades. Indeed, recent 
interpretations of minimal standards suggest that predictive 
validity coefficients of about .30 are adequate. 

APS Reading Test , With the exception of English 010 - 
Reading and Study Skills, that had only a maximum of 10 
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students in the correlations, the median predictive validity 
value across all levels is .14, ranging from a low of -.61 
to a high of .44. The correlation indicates a low linear 
relationship between the placement test scores and course 
grades . 

The reader is reminded that these correlations apply 
only to the ability of the APS tests to predict the final 
course grade performance of first-time college students. 

CONCLUSIONS 

APS Writing Test . In light of the myriad of factors that 
impact course grades other than the specific skills measured 
by the test, and the unreliability of the criterion (course 
grades), the obtained value of the predictive validity of 
the APS Writing Test is adequate to justify use of the test 
for placing students into writing courses at College of the 
Canyons . 

APS Reading Test . The APS Reading Test has still not been 
shown to be a valid predictor of final course grades for 
reading courses at College of the Canyons. 

The college should continue to evaluate the APS Reading 
Test and multiple measures which may provide the most 
accurate advice possible when directing students into 
classes. Further predictive validity computations will be 
performed using Spring 1994 data, as it becomes available. 
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VALIDATING PLACEMENT RULES 
FOR THE APS WRITING TEST 

June 1994 

The report entitled Predictive Validity Study of the 
APS Writing and Reading Tests (May 1994) showed the adequate 
overall relationship between test scores and course grades 
for the APS Writing Test, supporting the use of the test 
scores as predictors of student performance at College of 
the Canyons • 

The next issue is a very practical one — how best to 
use the tests for making placement recommendations. This 
involves identifying cut scores. At what point in the 
continuum of test scores do we recommend placement of 
students in a given course? For example, are students who 
receive scores of 28 to 40 on the APS Writing Test fully 
prepared to undertake English 101 - English Composition and 
Literature? Should the score range be broader or narrower? 
No method of identifying cut scores is flawless. With 
typical validities of .35, only about 12 percent of the 
variance in student performance is associated with the test 
scores. A great many factors not tapped by the tests 
contribute to course grades. The goal of the process is to 
improve prediction of success. 
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METHOD 

The primary method used for establishing cut scores 
involves comparing success rates for contrasting groups over 
a range of scores. More specifically, the contrasting 
groups are defined as "successful students" or those who 
earned grades of A, B, C and CR, whereas persons with grades 
of D, F, NC or W (withdrawal) are categorized in the 
"unsuccessful'" group. Given this distinction, it is then 
possible to determine the proportion of students with 
specific scores, and hence for score ranges, that were 
successful. Cut scores are then evaluated in terms of their 
ability to "predict" success (i.e. maximize the difference 
between success rates for persons scoring above a value and 
those scoring below it). 

Subjects . The research participants were 340 students from 
spring and fall semesters 1993. The students were enrolled 
in English Oil (N=6), English 035 (N=45) , English 090 
(N=250), and English 101 (N=39). The students were 
first-time college students and had taken the APS Writing 
Test just prior to enrolling in the English course. 
Participating students were placed in English courses using 
existing placement procedures, including the use of 
placement test scores. End-of-course grades were used as 
the student performance criterion. 
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RESULTS 

Placement Rules . The scatter diagrams presented in Figures 
1 through 3 show the relationship between test scores and 
success in each target class. These provide an initial 
basis for establishing trial cut scores to minimize 
placement errors. On the scatter plots, a number represents 
one or more coincident data points while letters designate 
two-digit numbers, i.e. A=10, B=ll, C=12 etc. Withdrawals 
were treated as unsuccessful course outcomes in this 
analysis of dichotomized grades. 
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Figure 1. 

English 035 Success by APS Writing Test 
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Figure 2. 

English 090 Success by APS Writing Test 
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Figure 3 . 

English 101 Success by APS Writing Test 
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The scatterplots were partitioned using a variety of 
possible cut scores. The partitions produced placement 
classification tables for each trial cut score. The table 
partitions the data from the scatterplot into four 
quadrants. The upper right and lower left quadrants 
represent correct placements. The other two quadrants 
represent placement errors. 

The analysis showed that there were more than one set 
of cut scores for each course that were optimum in placement 
accuracy. The recommended cut scores shown in Table 1 were 
set in the lower part of the optimum range for each course. 



Table 1. 

Recommended APS Writir.g Test Cut Scores by Course 



Test 


Score 


Range 


Recommended 
Placement 


% of Study Sample 
in Score Range N 


Writing 


0 


- 12 


English Oil 


5.0% 17 




13 


- 16 


English 035 


10.4% 35 




17 


- 23 


English 090 


41.3% 139 




24 


- 40 


English 101 


43.4% 146 
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Current APS Writing Test Cut Scores By Course 



Test 


Score Range 


Recommended 
Placement 


% of Study Sample 
in Score Range N 


Writing 


0-10 


English Oil 


1.2% 4 




11 - 15 


English 035 


12.4% 42 




16 - 27 


English 090 


74.8% 252 




28 - 40 


English 101 


11.6% 39 



Placement Accuracy . Tables 2 through 4 are placement 
classification tables that show the relationships between 
admission status (that would have resulted from use of the 
recommended placement rules) and actual student performance 
observed in the course. For this analysis, students were 
considered eligible for the course they selected if their 
score exceeded the cut score, even if that score would have 
qualified them for admission to a higher level course. 
Correct classifications are represented in the upper right 
and lower-left cells of the table, the sum of these cells 
represents the total of correct placement. The percentage 
of correct classifications is shown as a footnote to each 
table. In addition, the footnotes show the net gain in 
correct placements that would result from use of the 
recommended cut score and the percentage of students 
eligible (selection ratio) under this placement rule. 
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Table 2. 
Percentage of Students by 
English 035 Success and Admission status 

CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 

Database: English 02 - S4F93 (Filter: NONE) 

Course: ENGL 035 

Test: ENGLISH 02: APS WRITING A 

Subgroup: TOTAL (N = 45 ) 



NOT ELIG ELIGIBLE 
< 13 >= 13 

















Correct Placements : 


57 


8% 








3 




18 


21 


Base Rate(non-succ): 


53 


3% 


SUCCESS 




6 


. 7% 




40 .0% 


46.7% 


Gain in Accuracy: 


4 


4% 


A.B.CCR 


r 


14 


. 3% 


r 


85 . 7% 


r 100.0% 










c 


27 


.3% 




o 2 . 9% 
























Selection Ratio: 


75 


6% 








8 




16 


24 


Diff in Succ Ratio: 


25 


7% 


NOT Sl'CC 




17 


.8". 




35 . 6% 


53 . 3% 








D.F.NC.W 


r 


33 


. 3% 


r 


66 . 7% 


r 100.0% 










c 


72 


. 7% 


c 


47. 1% 
















11 




34 


45 












24 


.4% 




75 . 6% 


100 .0% 









c 100.0% c 100.0% 



Table 3. 
Percentage of Students by 
English 090 Success and Admission Status 

CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 
Database: English 02 - S&F93 (Filter: NONE) 
Course: ENGL 090 

Test: ENGLISH 02: APS WRITING A 
Subgroup: TOTAL (N = 244) 



NOT ELIG ELIGIBLE 
< 17 >= 17 















Correct Placements: 


64 


3% 






3 




149 


152 


Base Rate (success): 


62 


3% 


SUCCESS 


1 


.2% 




61 . 1% 


62 . 3% 


Gain in Accuracy: 


2 


0% 


A.B.C , CR 


r 2 


.0% 


r 


98.0% 


r 100.0% 










c 27 


. 3% 


c 


63 . 9% 






















Selection Ratio: 


95 


5% 






8 




84 


92 


Diff in Succ Ratio: 


36 


7% 


NOT SUCC 


3 


.3% 




34 .4% 


37.7% 








D.F.NC.W 


r 8 


. 7% 




91 .3% 


r 100.0% 










c 72 


. 7% 


c 


36 . 1% 














11 




233 


244 










4 


.5% 




95.5% 


100 .0% 










c 100 


.0% 


c 


100.0% 
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Table 4. 
Percentage of Students by 
English 101 Success and Admission Status 



CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 
Database: English 02 - S&F93 (Filter: NONE) 
Course: ENGL 101 

Test: ENGLISH 02: APS WRITING A 
Subgroup: TOTAL IN = 38) 



NOT ELIG ELIGIBLE 
< 24 >= 24 















Correct Placements : 


65 






2 




22 


24 


Base Rate (success) 


63 


SUCCESS 


5 


3% 




57.9% 


63.2% 


Gain in Accuracy: 


2 


A.B.C.CR 


r 8 


3% 


r 


91.7% 


r 100.0% 








c 40 


0% 


c 


66.7% 




















Selection Ratio: 


86 






3 




11 


14 


Diff in Succ Ratio: 


26 


NOT SUCC 


7 


9% 




2 8.9% 


36 .8% 






D,F,NC,W 


r 21 


AX 


r 


78 . 6% 


r 100. C% 








c 60 


0% 


c 


33 . 3% 












5 




33 


38 








13 


. 2% 




86.8% 


100.0% 








c 100 


. 0% 


c 


100 .0% 









The following line graphs provide a visual depiction of 
three of the major factors considered when selecting the 
appropriate cut scores: (1) percentage of correct 
placements; (2) percentage of false positives - or those who 
were eligible but who received an unsuccessful final grade; 
and (3) percentage of false negatives - or those who were 
ineligible but who received a successful final grade. One 
common error in selecting cut scores is to only look at the 
the percentage of correct placements. By doing so, it 
encourages cut scores that are too high. Figures 4 through 
6 show the recommended cut scores for English 035, 090 and 
101. 
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Figure 4. 

English 035 Recommended Cut Score of 13 
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Figure 5. 

English 090 Recommended Cut Score of 17 
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Figure 6. 

English 101 Recommended Cut Score of 24 
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Table 5 provides supplemental information in support of 
the relationship between test scores and grades. 
Specifically, the mean test scores for key criterion (grade) 
defined groups — for students earning an A or B grade, a C 
or Credit grade, a grade of D, F or No Credit, and those 
receiving Ws — are presented. English Oil, 035 and 090 are 
all graded on a Credit/No Credit grading system, so grades 
of A and B were not present. The mean test scores for the 
successful students is clearly greater than the mean values 
for students who earned unsuccessful grades of D, F or No 
Credit. While the pattern is not a perfect linear 
relationship, these data are consistent with the overall 
linear relationships between test scores and course grades, 
and also support the reasonableness of the recommended cut 
score levels. 
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Table 5. 

Mean Score for Key Grade-Based Criterion Groups, 

By Course 



Final Grade Earned 
Test/Course A/B C/CR D/F/NC W 



APS Writing Test 










English Oil 






* 


* 


English 035 


★ 


16.67 


13.40 


15.89 


English 090 




23.28 


21.23 


22.91 


English 101 


32.00 


27.58 


30. 17 


27.50 


* N is 5 or less. 











Since the study sample represents only a portion of the 
students who actually took the APS Writing Test in Spring 
and Fall 1993, we questioned what proportion of the total 
group (N = 1498) would have fallen into each of the cut 
score ranges. Table 6 provides the breakdown. 
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Table 6. 

Recommended APS Writing Test Cut Score by Course, 
Spring and Fall 1993 



Test 


Score Range 


Recommended 
Placement 


% of 
Total Tested 


N 


Writing 


0-12 


English Oil 


4.2% 


63 




13 - 16 


English 035 


8.8% 


131 




17 - 23 


English 090 


30.9% 


464 




24 - 40 


English 101 


56. 1% 


840 



A smaller percentage of the total group of test-takers 
fell into each of the three lower cut score ranges than was 
true of the study sample. Using the recommended cut scores 
of 24-40, the majority (56.1%) of all tested students 
would have been recommended for placement in English 101. 
This compares to only 26.6% of the total group - had the 
current cut scores been used (28 - 40). 

DISCUSSION 

The findings of this study validate the use of the APS 
Writing Test to support placement decisions for students 
entering English 035, 090 and 101. Use of the test together 
with the proposed placement rules could increase the rate of 
correct placement over the base rate by 4.4% for English 
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035, by 2.0% for English 090 and 2.6% for English 101. 

A thorough review of these findings by all involved 
faculty and Student Services personnel is encouraged. 

CUTAPSW. FW4 
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